PDQ ATS research system
- Users new to R
- TAQ data assembled into partial books
- Running in Amazon EC2
- Research focused on transaction costs
Support diverse exploratory research programs
Running across all tickers from days to months or so
Lightweight software footprint, easy to use
You know how this starts
- Some data, maybe a lot of data
- Maybe a database
- Complicated modeling and research; need R or Python to be productive
- Then, maybe throw even more analysis tools into the mix
- You find yourself adding more computing horsepower
- Before you know it, you end up with…
Eventually, some of those arrows connect to R
Moving data around data centers not as big of a deal as you might think
http://www.eecs.berkeley.edu/~ganesha/disk-irrelevant_hotos2011.pdf
Cache/fast storage reuse is usually more important.
Marshaling those data in and out of various data formats is a big deal!
Yuck!
Feather: a common data frame serialization format for R and Python and …
SciDB streaming API
Databases a great at managing mutable data
Append-only data are much simpler to work with
Feathercache: a fast object store interface for R (experimental)
Simple GET/PUT/DELETE-like interface
Elastic computing with R and Redis on Amazon EC2
doRedis update due to CRAN soon, for now use GitHub
Vignettes
Amazon EC2 Recipe
- Instantiate a Linux image
- Install R and desired packages, including doRedis
- Run the doRedis service installer script
- Make an AMI from this instance
- Re-launch from AMI as needed, supplying config as EC2 user data